Skip to content

[RUM-12600] Continuous Benchmarking #3927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[RUM-12600] Continuous Benchmarking #3927

wants to merge 1 commit into from
+668 −2

Conversation

@amortemousque
Copy link
Collaborator

@amortemousque amortemousque commented Oct 23, 2025 •
edited
Loading

Motivation

We want to monitor the impact of performance-intensive features like Replay and Profiling.
This PR introduces benchmark tests to measure their effect.

Measurements

  • Memory, CPU, and network using the Chrome DevTools Protocol. Same implementation as /performances, except it only collects the total consumption.
  • Web Vitals using Google’s web-vitals package

Scenarios

Currently, we run a single scenario, heavy.scenario.ts, which executes a sample app from this PR.
The scenario doesn’t yet represent a realistic heavy website and will be refined in a follow-up PR.

This scenario runs under four different configurations:

  • none: no SDK
  • none_with_header: SDK disabled but profiling headers present
  • rum: RUM SDK only
  • rum_replay: RUM with Session Replay
  • rum_profiling: RUM with Profiling

Scheduling

A new performance-benchmark GitLab job runs every 30 minutes.
Each run executes the tests 15 times.

Visualization

Local: displays a summary table in the console
image

Datadog dashboard
image

Changes

Benchmark tests are in /benchmark. I haven’t replaced /performances yet, but if we agree on this approach, I’ll remove it in a follow-up PR.

Test instructions

yarn test:performance

Checklist

  • Tested locally
  • Tested on staging
  • Added unit tests for this change.
  • Added e2e/integration tests for this change.

@datadog-official
Copy link

datadog-official bot commented Oct 23, 2025 •
edited by datadog-datadog-prod-us1 bot
Loading

⚠️ Tests

⚠️ Warnings

🧪 1 Test failed

recorder › scroll positions › should be recorded across navigation from recorder/recorder.scenario.ts (Datadog) 
createTest.ts:244:3 should be recorded across navigation

[Firefox] › ../lib/framework/createTest.ts:244:3 › recorder › scroll positions › should be recorded across navigation 

    Error: expect(received).toHaveLength(expected)

    Expected length: 2
    Received length: 1
    Received array:  [{"application": {"id": "37fe52bf-b3d5-4ac7-ad9b-44882d479ec8"}, "creation_reason": "init", "end": 1762777775409, "has_full_snapshot": true, "index_in_view": 0, "records": [{"data": {"height": 720, "href": "http://172.28.166.18:9246/", "width": 1280}, "timestamp": 1762777775409, "type": 4}, {"data": {"has_focus": true}, "timestamp": 1762777775409, "type": 6}, {"data": {"initialOffset": {"left": 0, "top": 150}, "node": {"childNodes": [{"id": 1, "name": "html", "publicId": "", "systemId": "", "type": 1}, {"attributes": {"rr_scrollTop": 150}, "childNodes": [{"attributes": [Object], "childNodes": [Array], "id": 2, "tagName": "head", "type": 2}, {"id": 3, "textContent": "
          ", "type": 3}, {"attributes": [Object], "childNodes": [Array], "id": 15, "tagName": "body", "type": 2}], "id": 16, "tagName": "html", "type": 2}], "id": 17, "type": 0}}, "timestamp": 1762777775409, "type": 2}, {"data": {"height": 720, "offsetLeft": 0, "offsetTop": 0, "pageLeft": 0, "pageTop": 150, "scale": 1, "width": 1280}, "timestamp": 1762777775409, "type": 8}], "records_count": 4, "session": {"id": "8aea6db9-3ddb-4127-ba5c-d9a78d14b1ec"}, "source": "browser", "start": 1762777775409, "view": {"id": "191e73b0-a8af-4fc8-8400-5766e36c26c3"}}]
...

ℹ️ Info

❄️ No new flaky tests detected

🎯 Code Coverage
• Patch Coverage: 100.00%
• Total Coverage: 92.63% (+0.00%)

View detailed report

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 915b4f4 | Docs | Datadog PR Page | Was this helpful? Give us feedback!

@amortemousque amortemousque force-pushed the aymeric/profiling-benchmark branch from f8d98a3 to 036f257 Compare October 23, 2025 13:47
@amortemousque amortemousque changed the title Continuous Benchmark Continuous Benchmarking Oct 23, 2025
@cit-pr-commenter
Copy link

cit-pr-commenter bot commented Oct 23, 2025 •
edited
Loading

Bundles Sizes Evolution

📦 Bundle Name Base Size Local Size 𝚫 𝚫% Status
Rum 163.88 KiB 163.88 KiB 0 B 0.00% ✅
Rum Profiler 4.84 KiB 4.84 KiB 0 B 0.00% ✅
Rum Recorder 19.78 KiB 19.78 KiB 0 B 0.00% ✅
Logs 56.35 KiB 56.35 KiB 0 B 0.00% ✅
Flagging 944 B 944 B 0 B 0.00% ✅
Rum Slim 121.13 KiB 121.13 KiB 0 B 0.00% ✅
Worker 23.63 KiB 23.63 KiB 0 B 0.00% ✅
🚀 CPU Performance
Action Name Base CPU Time (ms) Local CPU Time (ms) 𝚫%
RUM - add global context 0.0042 0.0061 +45.24%
RUM - add action 0.0131 0.013 -0.76%
RUM - add error 0.0118 0.0125 +5.93%
RUM - add timing 0.0028 0.0032 +14.29%
RUM - start view 0.0034 0.0041 +20.59%
RUM - start/stop session replay recording 0.0007 0.0011 +57.14%
Logs - log message 0.0137 0.0298 +117.52%
🧠 Memory Performance
Action Name Base Memory Consumption Local Memory Consumption 𝚫
RUM - add global context 25.72 KiB 26.54 KiB +836 B
RUM - add action 48.05 KiB 47.98 KiB -76 B
RUM - add timing 24.76 KiB 23.78 KiB -1004 B
RUM - add error 52.95 KiB 53.40 KiB +465 B
RUM - start/stop session replay recording 24.18 KiB 23.63 KiB -569 B
RUM - start view 423.43 KiB 429.63 KiB +6.20 KiB
Logs - log message 42.49 KiB 43.48 KiB +1007 B

🔗 RealWorld

BenoitZugmeyer
BenoitZugmeyer reviewed Oct 24, 2025
View reviewed changes
benchmark/scenarios/heavy.scenario.ts Outdated
Comment on lines 4 to 5
test.describe('benchmark', () => {
createBenchmarkTest('heavy').run(async (page, takeMeasurements) => {
Copy link
Member

@BenoitZugmeyer BenoitZugmeyer Oct 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💭 thought: ‏I think there are more idiomatic ways to declare common test logic in playwright using fixtures. But personally I don't mind your approach, as I find it more explicit than using playwright magic.

@amortemousque amortemousque changed the title Continuous Benchmarking [RUM-12600] Continuous Benchmarking Nov 4, 2025
@amortemousque amortemousque force-pushed the aymeric/profiling-benchmark branch 5 times, most recently from 4a7b425 to 9d4e4fb Compare November 10, 2025 11:20
@amortemousque amortemousque marked this pull request as ready for review November 10, 2025 11:35
@amortemousque amortemousque requested a review from a team as a code owner November 10, 2025 11:35
@amortemousque amortemousque force-pushed the aymeric/profiling-benchmark branch 2 times, most recently from 9f651bf to 0fb5c4c Compare November 10, 2025 11:48
@amortemousque amortemousque force-pushed the aymeric/profiling-benchmark branch from 0fb5c4c to 915b4f4 Compare November 10, 2025 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BenoitZugmeyer BenoitZugmeyer BenoitZugmeyer left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@amortemousque @BenoitZugmeyer